Learning Music-Dance Representations through Explicit-Implicit Rhythm Synchronization

Although audio-visual representation has been proven to be applicable in many downstream tasks, the of dancing videos, which is more specific and always accompanied by music with complex auditory contents, remains challenging uninvestigated. Considering intrinsic alignment between cadent movement dancer rhythm, we introduce MuDaR , a novel xmlns:xlink="http://www.w3.org/1999/xlink">Mu sic- xmlns:xlink="http://www.w3.org/1999/xlink">Da nce xmlns:xlink="http://www.w3.org/1999/xlink">R epresentation learning framework perform synchronization dance rhythms both explicit implicit ways. Specifically, derive based on visual appearance motion cues inspired rhythm analysis. Then are temporally aligned counterparts, extracted amplitude sound intensity. Meanwhile, exploit coherence implied audio streams contrastive learning. The model learns joint embedding predicting temporal consistency pairs. music-dance representation, together capability detecting rhythms, can further applied three tasks: (a) classification, (b) retrieval, (c) retargeting. Extensive experiments demonstrate that our proposed outperforms other self-supervised methods large margin.